Moreover,compellingevidenceshowsthat GNNs are extremely vulnerable to graph data shifts [79,26,20].

Note that the shift components can be customized and expanded based on downstream tasks.

Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents' analysis approaches.

Feature engineering has demonstrated substantial utility for many machine learning workflows, such as in the small data regime or when distribution shifts are severe. Thus automating this capability can relieve much manual effort and improve model performance. Towards this, we propose AutoMAN, or Automated Mask-based Feature Engineering, an automated feature engineering framework that achieves high accuracy, low latency, and can be extended to heterogeneous and time-varying data. AutoMAN is based on effectively exploring the candidate transforms space, without explicitly manifesting transformed features. This is achieved by learning feature importance masks, which can be extended to support other modalities such as time series. AutoMAN learns feature transform importance end-to-end, incorporating a dataset's task target directly into feature engineering, resulting in state-of-the-art performance with significantly lower latency compared to alternatives.

Graph data are inherently complex and heterogeneous, leading to a high natural diversity of distributional shifts. However, it remains unclear how to build machine learning architectures that generalize to complex non-synthetic distributional shifts naturally occurring in the real world. Here we develop GraphMETRO, a Graph Neural Network architecture, that reliably models natural diversity and captures complex distributional shifts. GraphMETRO employs a Mixture-of-Experts (MoE) architecture with a gating model and multiple expert models, where each expert model targets a specific distributional shift to produce a shift-invariant representation, and the gating model identifies shift components. Additionally, we design a novel objective that aligns the representations from different expert models to ensure smooth optimization. GraphMETRO achieves state-of-the-art results on four datasets from GOOD benchmark comprised of complex and natural real-world distribution shifts, improving by 67% and 4.2% on WebKB and Twitch datasets.

Whenever you work on a deep learning project with Pytorch, you might have to define your dataset class at some point. In my case, I work on a project using semantic segmentation to train a transformer model that can generalize geometric shapes (such as building footprints) on different scales. When working out my implementation, I found it hard to find specific examples treating semantic segmentation, which is why I decided to share some parts of my experience. In the following, I will show you how I set up my dataset class object and apply the wanted data transformations to the "input" and the "mask" data. Let's go through this code step by step.

It appears that every sophisticated ML team has built a feature store for their ML platform. Google Cloud worked with our customer GoJek to build Feast. Fortunately, you no longer need to build or manage your own. Google Cloud Vertex AI offers a fully managed feature store as does Sagemaker. There are even companies like tecton.ai

Text models currently struggle to produce factually correct responses when asked questions about factual information. They can generate responses that appear compelling to laymen but are entirely incorrect. This problem is attributed to the current challenges in natural language processing which include contextual words, homonyms, synonyms, sarcasm, and ambiguity. The negative societal implications of text generation are fake news and the spread of disinformation. It can be used to produce believable content on a massive scale with much greater efficiency and lower barriers to entry. It can also be used for harmful activities like spam, phishing, abuse of legal and government processes, fraudulent academic writing, and pretexting.

It is common in statistics and machine learning to create a linear transform or mapping of a variable. An example is a linear scaling of a feature variable. We have the natural intuition that the mean of the scaled values is the same as the scaled value of the mean raw variable values. Unfortunately, we bring this intuition with us when using nonlinear transformations of variables where this relationship no longer holds. Fixing this intuition involves the discovery of Jensen's Inequality, which provides a standard mathematical tool used in function analysis, probability, and statistics.

Stochastic gradient Langevin dynamics (SGLD) is a widely used sampler for the posterior inference with a large scale dataset. Although SGLD is designed for unbounded random variables, many practical models incorporate variables with boundaries such as non-negative ones or those in a finite interval. Existing modifications of SGLD for handling bounded random variables resort to heuristics without a formal guarantee of sampling from the true stationary distribution. In this paper, we reformulate the SGLD algorithm incorporating a deterministic transformation with rigorous theories. Our method transforms unbounded samples obtained by SGLD into the domain of interest. We demonstrate transformed SGLD in both artificial problem settings and real-world applications of Bayesian non-negative matrix factorization and binary neural networks.